Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Event Log] Adding type_id to saved object array in event log #100939

Merged
merged 16 commits into from
Jun 4, 2021
Merged

[Event Log] Adding type_id to saved object array in event log #100939

merged 16 commits into from
Jun 4, 2021

Conversation

ymao1
Copy link
Contributor

@ymao1 ymao1 commented May 28, 2021 •
edited
Loading

Resolves #95411

Summary

Added type_id to saved object array field inside the event log. This captures the ruleTypeId for an alert saved object and the actionTypeId for an `action saved object.

Checklist

Delete any items that are not applicable to this PR.

@ymao1
Copy link
Contributor Author

ymao1 commented Jun 1, 2021

@elasticmachine merge upstream

@ymao1 ymao1 self-assigned this Jun 1, 2021
@ymao1 ymao1 added Feature:EventLog release_note:skip Skip the PR/issue when compiling release notes Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) v7.14.0 v8.0.0 labels Jun 1, 2021
@ymao1 ymao1 marked this pull request as ready for review June 1, 2021 12:42
@ymao1 ymao1 requested a review from a team as a code owner June 1, 2021 12:42
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

@pmuellr
Copy link
Member

pmuellr commented Jun 1, 2021

Would we want to fold some of this in as well? #94137 (comment) ?

If we didn't have to worry about actions, we could just populate those rule. fields instead of the new primary_saved_object field, but ... of course we can't, so it does seem like the primary_saved_object field will be handy.

It would be good to try to capture the action type id as well.

Also, would it be useful to try to do one of those "copy" mappings? So we can take the source field once, but use it in > 1 elasticsearch fields? Not quite sure how they work ...

@ymao1
Copy link
Contributor Author

ymao1 commented Jun 2, 2021 •
edited
Loading

It would be good to try to capture the action type id as well.

@pmuellr Good idea! Added in this commit: f99ecd0

Also, would it be useful to try to do one of those "copy" mappings? So we can take the source field once, but use it in > 1 elasticsearch fields? Not quite sure how they work ...

It seems like the utility of copy_to (docs) is to combine multiple fields into a single searchable field when you may not have control of the input. I feel like since we have control of what's being written, we should be making sure to just write out what we want to search on. I'm also not sure how feasible it is to use copy_to to copy from or to a nested field mapping.

@ymao1
Copy link
Contributor Author

ymao1 commented Jun 2, 2021

After some discussion with @pmuellr and @YulNaumenko, we will be adding the a type_id field to the nested saved objects array (either rule type id for alert SOs or action type id for action SOs) and removing the primary_saved_object field. @pmuellr has had success with defining runtime fields against the SO array in the event log so we will explore that option before making unnecessary copies of data within the event log.

@ymao1 ymao1 changed the title [Event Log] Adding rule_type_id and primary_saved_object to event log [Event Log] Adding type_id to saved object array in event log Jun 2, 2021
@ymao1
Copy link
Contributor Author

ymao1 commented Jun 2, 2021

After some discussion with @pmuellr and @YulNaumenko, we will be adding the a type_id field to the nested saved objects array (either rule type id for alert SOs or action type id for action SOs) and removing the primary_saved_object field. @pmuellr has had success with defining runtime fields against the SO array in the event log so we will explore that option before making unnecessary copies of data within the event log.

Done in this commit: 3b8b3fa

@ymao1 ymao1 requested a review from pmuellr June 2, 2021 17:27
YulNaumenko
YulNaumenko approved these changes Jun 3, 2021
View reviewed changes
Copy link
Contributor

@YulNaumenko YulNaumenko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

pmuellr
pmuellr approved these changes Jun 4, 2021
View reviewed changes
Copy link
Member

@pmuellr pmuellr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

I happened to think - "I wonder if we'll have a problem when users transition from < 7.14 to >= 7.14, and that field doesn't exist?"

I think for this code, everything seems fine - we're passing the value in, and not depending on it being set anywhere in records we're reading from. Could potentially be a problem in the future, if we forget that and are doing searches across different versioned event logs, expecting it to be set. But given the TS typing on these ends up being string | undefined, I think we'll be fine - at least TS type-wise (for example, getting an unexpected null dereference at runtime).

Just bringing that up in case anyone happens to think of other cases where that could be a problem, but I think we're fine.

@ymao1
Copy link
Contributor Author

ymao1 commented Jun 4, 2021

@elasticmachine merge upstream

@ymao1 ymao1 added the auto-backport Deprecated - use backport:version if exact versions are needed label Jun 4, 2021
@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @ymao1

@ymao1 ymao1 merged commit c13ae7e into elastic:master Jun 4, 2021
kibanamachine added a commit to kibanamachine/kibana that referenced this pull request Jun 4, 2021
@ymao1 @kibanamachine
[Event Log] Adding type_id to saved object array in event log (elas…
4033c91 
…tic#100939)

* Adding new fields to event log mapping

* Populating new event log fields when executing rules and actions

* Fixing functional tests

* Adding actionTypeId

* Putting type ids into saved object array

* Fixing functional tests

* Cleanup

Co-authored-by: Kibana Machine <[email protected]>
@kibanamachine kibanamachine mentioned this pull request Jun 4, 2021
Merged
@kibanamachine
Copy link
Contributor

💚 Backport successful

Status Branch Result
✅ 7.x

This backport PR will be merged automatically after passing CI.

kibanamachine added a commit that referenced this pull request Jun 5, 2021
@kibanamachine @ymao1
[Event Log] Adding type_id to saved object array in event log (#100939
9182941 
) (#101444)

* Adding new fields to event log mapping

* Populating new event log fields when executing rules and actions

* Fixing functional tests

* Adding actionTypeId

* Putting type ids into saved object array

* Fixing functional tests

* Cleanup

Co-authored-by: Kibana Machine <[email protected]>

Co-authored-by: ymao1 <[email protected]>
gmmorris added a commit to gmmorris/kibana that referenced this pull request Jun 7, 2021
@gmmorris
Merge branch 'master' into task-manager/capacity-estimation
e40b071 
* master: (90 commits)
  Fix UI breaks on providing long search keyword in 'Search Box' (elastic#101385)
  Adds css class to EuiDescriptionListDescription in order to break word on exception details card (elastic#101481)
  [Lens] Increase timings for drag and drop tests (elastic#101380)
  [Lens] Fix editor react error on configuration panel (elastic#101367)
  [Fleet] Move integrations to a separate app (elastic#99848)
  Fix incorrect message displayed on importing Timeline Templates (elastic#101288)
  [Cases] RBAC (elastic#95058)
  [APM] Visual improvements for new APM layout with left navigation (elastic#101360)
  [master] More precise alerts matching (elastic#99820)
  [Lens] Value in legend (elastic#101353)
  Revert "[Reporting] ILM policy for managing reporting indices (elastic#100130)" (elastic#101358)
  [Discover] Fix header row of data grid in Firefox (elastic#101374)
  Add link to advanced setting in Discover (elastic#101154)
  Url service locators (elastic#101045)
  [Timelion] Update the removal message to mention the exact version (elastic#100994)
  [Security Solution][Detection Engine] Test cases for alias failure test cases where we don't copy aliases correctly (elastic#101437)
  [Event Log] Adding `type_id` to saved object array in event log (elastic#100939)
  [Reporting] Add `location.url` info to console message logs (elastic#101427)
  [Security Solutions][Detection Engine] Fixes timestamp bugs within source indexes when the formats are not ISO8601 format (elastic#101349)
  Improve Task Manager instrumentation (elastic#99160)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@YulNaumenko YulNaumenko YulNaumenko approved these changes

@pmuellr pmuellr pmuellr approved these changes

Assignees

@ymao1 ymao1

Labels
auto-backport Deprecated - use backport:version if exact versions are needed Feature:EventLog release_note:skip Skip the PR/issue when compiling release notes Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) v7.14.0 v8.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[event log] add rule type id in custom kibana.alerting field
5 participants
@ymao1 @elasticmachine @pmuellr @kibanamachine @YulNaumenko